Aggregating Regression Procedures for a Better Performance

Methods have been proposed to linearly combine candidate regression procedures to improve estimation accuraccy. Applications of these methods in many examples are very succeesful, pointing to the great potential of combining procedures. A fundamental question regarding combining procedure is: What is the potential gain and how much one needs to pay for it? A partial answer to this question is obtained by Juditsky and Nemirovski (1996) for the case when a large number of procedures are to be combined. We attempt to give a more general solution. Under a l 1 constrain on the linear coeecients, we show that for pursuing the best linear combination over n procedures, in terms of rate of convergence under the squared L 2 loss, one can pay a price of order O ? log n=n 1? when 0 < < 1=2 and a price of order O (log n=n) 1=2 when 1=2 < 1. These rates can not be improved or essentially improved in a uniform sense. This result suggests that one should be cautious in pursuing the best linear combination, because one may end up with paying a high price for nothing when linear combination in fact does not help. We show that with care in aggregation, the nal procedure can automatically avoid paying the high price for such a case and then behaves as well as the best candidate procedure in terms of rate of convergence.